Joint Maximum Margin and Maximum Entropy Learning of Graphical Models

INFERRING structured predictions based on correlated covariates remains a central problem in many fields, including NLP, computer vision, and computational biology. Typically, both the input covariates and output predictions can be high-dimensional, multi-modal, noisy, partially observable, and bearing latent structures, each of these characteristics adds a degree of complexity to the task of learning structured input/output (I/O) models. Several recent approaches to structured I/O learning are based on learning discriminative probabilistic graphical models. By defining composite features that explicitly exploit the structured dependencies among input elements (e.g., words in a sentence) and among the interpretational outputs (e.g., part-of-speech tags), such models can produce semantically consistent predictions from complex inputs. However, how to train such models properly remains a highly contested issue. The two dominant paradigms for training such models are the maximum (conditional) likelihood estimation (MLE) [6], which leads to the well-known CRF, and the max-margin learning [9], [10], which leads to the MN. While both methods have enjoyed remarkable success and are widely used, they have a number of deficiencies, as we will discuss below. In this project, we introduce a new paradigm for learning structured I/O models, and graphical models in general, that conjoins and extends the merits of MLE and maxmargin learning while avoiding their shortcomings.